Content display controls based on environmental factors

ABSTRACT

Approaches are described for controlling and filtering the display of content by computing devices. In particular, a mobile computing device can adjust the playing of media content (e.g., audio, video, images, games, ringtones) according to situational appropriateness and other environmental factors occurring around the device. For example, a mobile computing device such as a smart phone could use image and audio information captured by its sensors to determine that minors are present within the vicinity of the mobile phone, and filter certain portions of media content that would be deemed inappropriate for those minors. The environmental information can be determined based on data gathered by various sensors of the mobile device, including but not limited to cameras, microphones, global positioning system (GPS) devices, light sensors and others.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/457,889, filed Mar. 13, 2017, which is a continuation of U.S. patentapplication Ser. No. 14/983,375, filed Dec. 29, 2015, now U.S. Pat. No.9,596,568, which is a continuation of U.S. patent application Ser. No.13/609,572, filed on Sep. 11, 2012, now U.S. Pat. No. 9,288,387, thecontents of which are entirely incorporated by reference herein.

BACKGROUND

In recent years, portable computing devices, such as tablets and smartphones, have been increasingly utilized by consumers for a number ofdifferent purposes. Besides making telephone calls and browsing websiteson the Internet, these devices are often used to render media (e.g.,audio and video), capture photographs, play games, manage personal tasksand perform a variety of other functions. In some cases, the portabledevice can be integrated with another device used to playback content,such as televisions, stereo equipment and the like. For example, amobile phone may transmit digital audio (e.g., MP3 content) to a stereosystem which can play the audio in higher sound quality than what themobile phone could otherwise achieve. Similarly, the device may providevideo or other image content to a television for display in the user'shome. All of this increased integration presents opportunities toprovide better, more tailored and granular control to the user over thevarious forms of content being displayed.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an example of an environment in which the mobilecomputing device is used to control content presenting, in accordancewith various embodiments;

FIG. 2 is an illustration of another environment in which a mobilecomputing device can control the presenting of media content, inaccordance with various embodiments;

FIG. 3 illustrates an example of a mobile computing device that usesfacial recognition to control the display of media content, inaccordance with various embodiments;

FIG. 4 illustrates an example of various components of the mobilecomputing device that can be used to control the display of mediacontent, in accordance with various embodiments;

FIG. 5 illustrates an example process for controlling the presenting ofmedia content based on environmental factors, in accordance with variousembodiments;

FIG. 6 illustrates front and back views of an example portable computingdevice that can be used in accordance with various embodiments;

FIG. 7 illustrates an example set of basic components of a portablecomputing device, such as the device described with respect to FIG. 6;and

FIG. 8 illustrates an example of an environment for implementing aspectsin accordance with various embodiments.

DETAILED DESCRIPTION

In the following description, various embodiments will be illustrated byway of example and not by way of limitation in the figures of theaccompanying drawings. References to various embodiments in thisdisclosure are not necessarily to the same embodiment, and suchreferences mean at least one. While specific implementations and otherdetails are discussed, it is to be understood that this is done forillustrative purposes only. A person skilled in the relevant art willrecognize that other components and configurations may be used withoutdeparting from the scope and spirit of the claimed subject matter.

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the foregoing or otherdeficiencies experienced in conventional approaches for displaying andotherwise presenting content via one or more computing devices. Inparticular, various embodiments provide approaches for filtering andotherwise controlling the presentation of content (e.g., audio, video,images, ringtones, games) by computing devices according to situationalappropriateness and other environmental factors occurring around thedevice. For example, a mobile computing device such as a smart phonecould use image data and/or audio information captured by its sensors todetermine that minors are present within the vicinity of the mobilephone, and filter certain portions of media content that would be deemedinappropriate for those minors. The environmental information can bedetermined based on data gathered by various sensors of the mobiledevice, including but not limited to cameras, microphones, globalpositioning system (GPS) devices, light sensors and others.

In some instances, instead of directly presenting the content, themobile device (e.g., mobile phone) can provide the media content toanother device for presentation while still controlling one or moreaspects of the content. By way of example, the mobile phone mayestablish a connection with a stereo system of an automobile and use thestereo system to play music or other audio inside the automobile. Whileplaying the music tracks, the device may determine that a child ispresent in the car and that the next song in the playlist includesprofanity or other inappropriate material. For example, the mobile phonecan use one or more cameras embedded therein to identify that an adultis with a child and then either skip the song in the playlist or filterthe inappropriate material from the song. In various embodiments, thedevice can briefly turn off the audio during the moments of profanity,“bleep out” the swear words, or otherwise censor the inappropriatematerial. The amount of filtering could be adjusted as the devicerecognizes various people in a family, perhaps applying more filteringfor a 5 year old child than a 13 year old teenager, based on a set ofparental rules. In some embodiments, the parental rules can beconfigurable by the owner of the device (e.g., parent).

By way of another example, when video is being played on the device orthrough a HDMI connection to a screen, the display can be adjusted byusing overlays to remove swearing or adult scenes when a child is in theroom (or just during the period of time that the child walks into theroom). In some embodiments, the underlying content does not need to bemodified. Instead, one or more filters can be applied that skip overcertain portions of scenes, substitute audio over swear words orotherwise censor the content portions.

In some embodiments, the device can switch to different modes based oncontextual clues such as identified persons in proximity of the device.For example, the device may switch into an “accessibility” mode if theuser is recognized as someone who is disabled, older or otherwise wouldbenefit from those features. The accessibility mode may cause the deviceto display larger icons, provide audio information or receive audioinstructions from the user, among other such options. Similarly, thedevice may determine that the person may need easier accessibility modeif the user is having trouble hitting obvious touch points to executecertain functionality that would typically be the primary focus of thosewho did not need the accessibility features. In various embodiments, anumber of other modes each associated with specific functionality, canalso be implemented for use with the device, such as a child mode,specific disability modes (blind, deaf, etc.), public mode, quiet modeand the like.

Aside from recognizing persons present in the vicinity of the device,the computing device can use other environmental information to modifyand control the presentation of content. The presentation of content caninclude displaying one or more images or one or more videos, playing oneor more audio files and presenting any other media content. For example,the mobile computing device can learn that a certain location is a workenvironment and adjust ringtones, images or other content displayed bythe device so that inappropriate material is not displayed in theworkplace of the user. This can also be performed based on environmentalattributes other than location information. In environments wherelocation data isn't available, the computing device may use contextualclues about the environment, including but not limited colors, shapes ortypes of objects identified by images captured using a camera of thedevice, identified persons within proximity of the device, and the like.For example, it may be useful for the mobile phone device to modify aringtone if the user is determined to be out to dinner with workcolleagues. In this embodiment, the device could automatically recognizethat the persons in proximity of the device are identified as workcolleagues and determine that existing rules about “work” should stillapply, despite the fact that the device is not in the work “location”anymore. Based on that determination, the device could automaticallychange the ringtone, without the user having to remember to manuallychange it themselves. In some embodiments, the content can also besecured so that any third party who picked up an unlocked device wouldonly be able to access limited content that was appropriate for ageneral audience. All work-related e-mail, personal documents, and othersensitive information can be made inaccessible to the third party. Inthis embodiment, the user interface can revert to generic unless theowner of the device is identified and overrides the generic mode.

In various embodiments, the device can perform image analysis on datareceived from one or more cameras to identify one or more persons in theroom. For example, the device can apply facial recognition algorithms toimages of a user when attempting to recognize the user. The device couldalso use multiple cameras to obtain depth information which can be usedto determine distance to various objects or persons. For example, thedevice can use image information captured by multiple cameras todetermine that someone other than the typical user was within viewingdistance such that they could read over the user's shoulder. Based onthat determination, the device can perform a number of actions, such asdimming the screen, blocking or filtering content that may have beendesignated private, or notifying the user to inform them that theirinformation is at risk. Distance information can be determined byconfiguring two or more cameras for stereo image processing. Whenmultiple cameras are used, distance information in the image can bedetermined by comparing the offset in the image with the known distancebetween the two cameras embedded in the device. This distanceinformation can be used for a variety of purposes, including the abilityto determine when third party persons are within viewing distance of thedevice, as described above. In some embodiments, the device can furtheranalyze the audio information captured by a microphone to identify oneor more users that are speaking in the room. For example, one or morevoice recognition algorithms can be applied by the device to identify orauthenticate the user based his or her voice patterns (e.g., pitch,timber). In at least one embodiment, the device can store a voiceprintor other unique information for each user in the family and associatethe voiceprint with known user characteristics, such as age, sex, tasteor preferences and the like.

In some embodiments, the mobile computing device or a fixed device in aroom, (e.g., a set top box, streaming media device, etc.) can allow theusers to specify preferences for various types of content that would berendered by the mobile device. For example, one user may indicate thatthey do not like a particular song and the mobile device can skip overthat song in the playlist whenever that user is in the room. Thesepreferences can be manually configured using the interface of the deviceor can be automatically obtained from another location, such as adifferent computing device, or the like. In one embodiment, the devicecan use audio and imaging data captured by its sensors to identify allof the users in the room (or other environment) and selectively controlthe content being presented according to the users currently in thevicinity.

FIG. 1 illustrates an example of an environment in which the mobilecomputing device is used to control content presentation, in accordancewith various embodiments. In the illustrated embodiment, an automobile12 is shown with two adults (14, 16) sitting in the front seat and twochildren or teenagers (18, 20) sitting in the rear seat. The passengeradult 16 is holding a mobile computing device 22, such as a cellularphone. The mobile computing device 22 can establish a connection withthe stereo system 24 of the automobile and use the stereo system 24 toplay various media content (e.g., music) to the persons in the car. Theconnection can be either a wired connection (e.g., RCA cable) or awireless connection (e.g., Bluetooth, RF) through which the mobiledevice 22 can relay the audio information to the stereo system 24.

In accordance with an embodiment, the mobile computing device 22 can usedata gathered by its various sensors to determine that the persons inthe automobile include children or minors (18, 20). For example, themobile device can use image data captured by the front-facing andrear-facing cameras to identify the persons present in the vehicle usingvarious facial recognition techniques. As an alternative example, themobile device can use audio information captured by a microphoneembedded in the device to identify the persons based at least in part onvoice recognition software. In addition, the device may use otherinformation gathered by its various sensors to determine attributesabout its environment, including GPS information to determine that thedevice is in an automobile (i.e. traveling fast), information indicatingthat a connection has been established with the stereo system of theautomobile, and the like, in order to evaluate the environment in whichthe device is currently in. In some embodiments, the mobile device mayobtain some of the environmental information by establishingcommunication with the computing system of the automobile itself. Forexample, the main computer of the automobile may contain informationindicating that there are persons in the backseat weighing less than 70lbs. (e.g., children) or other useful environmental information whichcould be provided to the device.

In accordance with an embodiment, the mobile computing device 22controls the rendering of the media content based at least in part onthe determined environmental attributes, such as the presence ofchildren within the automobile, as illustrated in FIG. 1. For example,if the mobile device 22 is playing a playlist that is audible to thechildren (18, 20) sitting in the backseat, the device 22 may skip thesongs in the playlist that contain profanity, sexual references or otherexplicit material that the parents may have deemed inappropriate fortheir children. Alternatively, the mobile device may apply one or morefilters that silence or “bleep” the inappropriate material in the audiotrack. The song may be determined to contain the inappropriate materialbased on various ratings, such as the parental advisory (PAL) labelassociated with the song in the playlist.

In some embodiments, the amount or degree of filtering can be adjustedbased on the specific persons that have been identified to be presentwithin the automobile. For example, if the mobile device determines thatthere is a 5 year old present in the automobile, the device may entirelyskip certain audio tracks, while if the child is 13 years old, thedevice may simply censor the portions containing the profanity.

In accordance with an embodiment, the controlling and filtering of themedia content is configurable on the mobile device. For example, thedevice can be installed with an application that manipulates the varioussettings for the media content that will be displayed for the device. Insome embodiments, the device can store profiles (e.g., voiceprints,facial features, body attributes) of various users known by the owner ofthe device, such as the family members. The device can use theseprofiles to later identify which persons are present in the environmentaround the mobile device and adjust the display of content according tothese persons. In some embodiments, the device may further include adefault profile for strangers (i.e. people that are not able to beidentified by the device) which specifies how content is to be renderedwith strangers present in the vicinity of the device.

FIG. 2 is an illustration of another environment in which a mobilecomputing device can control the presenting of media content, inaccordance with various embodiments. It should be understood thatthroughout this disclosure, reference numbers for like components may becarried over between figures for purposes of explanation, but thisshould not be interpreted as limiting the scope of the variousembodiments.

In the illustrated embodiment, an environment of a living room is shownwith several persons sitting on a couch in front of a television 28.Similarly to the previous figure, the persons include parents (14, 16)and a child 20 sitting within a predetermined vicinity of the mobiledevice 26. In this example, the mobile device 26 is a tablet computerthat is being hand-held by the parent 14. For example, the parent 14 maybe using the device 26 to display media content, such as video ormultimedia. In one embodiment, the video may be displayed by the devicetransmitting the video to the television 28 in order to provide a largerdisplay area.

In accordance with an embodiment, the device 26 can detect the presenceof the child 20 within the vicinity of the device. The vicinity can beany predetermined area around the device, such as area within audible orviewable distance of the device 26 (or television 28), area within thesame room or other physical location as the device 26, or a specifieddistance away from the device 26. In various embodiments, the device canuse data received from the camera, microphone, GPS device or otherlocation sensor, light sensor, or other sensor embedded in the device.

In accordance with an embodiment, the device 26 may analyze theinformation retrieved from the various sensors and control or adjust theplayback or display of video or other media content by the device. Forexample, the device may use overlays to remove swearing or adult scenesfrom the video content as long as the child 20 is sitting within viewingdistance of the display. Alternatively, the device 26 may display awarning to the person 14 operating the device, indicating that anexplicit scene may be coming up which may be inappropriate for the childin the room. In some embodiments, once the child exits the room, thedevice can automatically adjust the display of media content to ceaseoverlaying the adult scenes.

FIG. 3 illustrates an example of a mobile computing device that usesfacial recognition to control the display of media content, inaccordance with various embodiments. In the illustrated embodiment, thechild 20 is operating the mobile device 32 in order to play music orother audio. This may be performed by the child 20 using the touchscreen 36 to select and play the songs from a playlist which areappealing to her. In some cases, the songs or other content may have aparental advisory label or other metadata indicating that the musiccontains inappropriate material.

In accordance with an embodiment, the device 32 may use the front-facingcamera 34 to capture an image 30 of the user 20 using the device. Thedevice can then apply some image recognition and facial recognitionalgorithms to identify the user that is currently operating the device32. Facial recognition, as well known in the art, can be any softwareand/or hardware to identify or verify a person from a digital image orvideo frame, such as by comparing a number of features extracted fromthe image against a database of other features. In this particularexample, the facial recognition application may match the user 20 to astored profile of the child 20 which may have been configured by theparent owner of the device. In other embodiments, however, the devicemay analyze the facial features of the user 20 in order to determine orestimate the approximate age of the user.

In accordance with the illustrated embodiment, the device 32 can thencontrol, filter and otherwise modify the playing of the media content(e.g., music, video, images) based on the identification of the user.For example, because the user 20 may be identified as being a youngchild, the device 32 may sensor various obscene material from the musicor video being displayed by the device.

In various embodiments described herein, besides identifying persons inthe vicinity of the device, the mobile computing device can also utilizethe data captured by its sensors to determine a variety of otherinformation about the physical environment around the device. Forexample, the device can utilize GPS information to determine that thedevice is at the user's work environment and therefore restrict thepresentation of various content in response to that determination.Alternatively, the device may simply determine that the device is in apublic location (e.g., restaurant, bar, shopping area) and control thepresentation of content accordingly. In some embodiments, the user mayconfigure the settings of the device to create profiles (e.g., a profilefor public locations, a profile for home environment, a profile for workenvironment, etc.). Each profile can indicate what types of contentshould be restricted, filtered or otherwise controlled by the mobiledevice. For example, the user may wish for the mobile device to playdifferent ringtones or turn off the sound of the device when the deviceis in the work environment versus the home environment.

FIG. 4 illustrates an example environment 400, showing variouscomponents of the mobile computing device that can be used to controlthe display of media content, in accordance with various embodiments. Inthe illustrated embodiment, the mobile computing device 401 contains anumber of sensors, including a global positioning system (GPS) receiver402, a camera 403, a microphone 404, and a light sensor 405. A GPSreceiver 402 can receive GPS signals for the purpose of determining thedevice's current location on Earth. The GPS device can provide latitude,longitude, and in some cases altitude information to the computingdevice 401. The camera 403 can be any device capable of capturing imagesor sequences of images (i.e. video). For example, a digital camera 403embedded into a mobile phone is typically comprised of a lens positionedin front of an image sensor that converts the optical image intoelectronic signal. The microphone 404 can be any device capable ofconverting sound into an electrical signal. The light sensor 405 can bea photoresistor or any other device capable of detecting light or otherelectromagnetic energy.

In various embodiments, the mobile computing device 401 can furtherinclude a content display filter and control component 406 and a contentplayer 407 capable of displaying media or other content 408. The contentplayer 407 can be any media player or other software or hardware that iscapable of rendering content to a person. In accordance with anembodiment, the content filter and control component 406 can use some orall of the data gathered by the sensors (402, 403, 404, 405), as well asother data, in order to determine one or more factors about theenvironment within the vicinity of the device. For example, the mobiledevice can use a combination of image data captured by the camera 403,sound captured by microphone 404 and location information provided bythe GPS receiver 402 in order to determine that the device is currentlyin the living room in which one or more children are present, aspreviously illustrated. Based on the analysis of this data, the contentfilter 406 can adjust the display of the content being played by thecontent player 407. For example, the filter 406 may censor at least aportion of audio or video being displayed, skip over the portion orotherwise adjust the rendering of the content 408.

In various embodiments, the content being displayed by the device 401can either be stored locally 408 on the device, or obtained from aremote location over a network, such as by using a network interfacecard 409. For example, the device may stream content from the “cloud”content provider that provides movies, music or other media content overa network. As used throughout this disclosure, a network can be anywired or wireless network of devices that are capable of communicatingwith each other, including but not limited to the Internet or other WideArea Networks (WANs), cellular networks, Local Area Networks (LANs),Storage Area Networks (SANs), Intranets, Extranets, and the like. Inaddition, the content being rendered by the content player 407 can bedisplayed on a user interface 410 of the device, transmitted to a remotedevice (e.g., stereo, television) or presented in some other manner asknown in the art.

FIG. 5 illustrates an example process 500 for controlling the presentingof media content based on environmental factors, in accordance withvarious embodiments. Although this figure may depict functionaloperations in a particular sequence, the processes are not necessarilylimited to the particular order or operations illustrated. One skilledin the art will appreciate that the various operations portrayed in thisor other figures can be changed, rearranged, performed in parallel oradapted in various ways. Furthermore, it is to be understood thatcertain operations or sequences of operations can be added to or omittedfrom the process, without departing from the scope of the variousembodiments. In addition, the process illustrations contained herein areintended to demonstrate an idea of the process flow to one of ordinaryskill in the art, rather than specifying the actual sequences of codeexecution, which may be implemented as different flows or sequences,optimized for performance, or otherwise modified in various ways.

In operation 501, a mobile computing device (e.g., cellphone, tabletcomputer, electronic reader) receives instructions to display content.The instructions can be received by the user activating a particularportion of the user interface of the device, by receiving an instructionprogrammatically from an application installed on the device or inanother way. The content can include audio, video, images, ringtone orother content perceptible by a user.

In operation 502, the device obtains data from one or more sensorsembedded in the device. For example, the device may obtain image datafrom one or more cameras, audio data from one or more microphones,location data from one or more location sensors (e.g., GPS receiver)and/or the like. In alternative embodiments, the mobile device obtainsthe data from another apparatus that is not embedded into the mobiledevice, such as by establishing communication with another computingdevice.

In operation 503, the device analyzes the data retrieved from thesensors in order to determine information about the persons within apredetermined vicinity of the device and/or other environmental factors.The determined information can include but is not limited toidentification of persons within the vicinity of the device,identification of the location in which the device is located, time ofday, or other situational awareness or appropriateness information.

In operation 504, the determined information is used to filter andcontrol the display of content by the device. For example, the devicemay censor or skip portions of video and/or audio, turn off ringtones,secure personal documents, or otherwise adjust the rendering of content.

FIG. 6 illustrates front and back views of an example portable computingdevice 600 that can be used in accordance with various embodiments.Although one type of portable computing device (e.g., a smart phone, anelectronic book reader, or tablet computer) is shown, it should beunderstood that various other types of electronic devices that arecapable of determining, processing, and providing input can be used inaccordance with various embodiments discussed herein. The devices caninclude, for example, notebook computers, personal data assistants,cellular phones, video gaming consoles or controllers, and portablemedia players, among others.

In this example, the portable computing device 600 has a display screen602 (e.g., a liquid crystal display (LCD) element) operable to displayimage content to one or more users or viewers of the device. In at leastsome embodiments, the display screen provides for touch or swipe-basedinput using, for example, capacitive or resistive touch technology. Sucha display element can be used to, for example, enable a user to provideinput by pressing on an area of the display corresponding to an image ofa button, such as a right or left mouse button, touch point, etc. Thedevice can also have touch and/or pressure sensitive material on otherareas of the device as well, such as on the sides or back of the device.While in at least some embodiments a user can provide input by touchingor squeezing such a material, in other embodiments the material can beused to detect motion of the device through movement of a patternedsurface with respect to the material.

The example portable computing device can include one or more imagecapture elements for purposes such as conventional image and/or videocapture. As discussed elsewhere herein, the image capture elements canalso be used for purposes such as to determine motion and receivegesture input. While the portable computing device in this exampleincludes one image capture element 604 on the “front” of the device andone image capture element 610 on the “back” of the device, it should beunderstood that image capture elements could also, or alternatively, beplaced on the sides or corners of the device, and that there can be anyappropriate number of capture elements of similar or different types.Each image capture element may be, for example, a camera, acharge-coupled device (CCD), a motion detection sensor, or an infraredsensor, or can utilize another image capturing technology.

The portable computing device can also include at least microphone(s)606, 608, and/or 612 or other audio capture element capable of capturingaudio data, such as may be used to determine changes in position orreceive user input in certain embodiments. In some devices there may beonly one microphone, while in other devices there might be at least onemicrophone on each side and/or corner of the device, or in otherappropriate locations.

The device 600 in this example also includes at least one motion orposition determining element 618 operable to provide information such asa position, direction, motion, or orientation of the device. Theseelements can include, for example, accelerometers, inertial sensors,electronic gyroscopes, electronic compasses, and GPS elements. Varioustypes of motion or changes in orientation can be used to provide inputto the device that can trigger at least one control signal for anotherdevice. The example device also includes at least one communicationmechanism 614, such as may include at least one wired or wirelesscomponent operable to communicate with one or more portable computingdevices. The device also includes a power system 616, such as mayinclude a battery operable to be recharged through conventional plug-inapproaches, or through other approaches such as capacitive chargingthrough proximity with a power mat or other such device. Various otherelements and/or combinations are possible as well within the scope ofvarious embodiments.

In order to provide functionality such as that described with respect toFIG. 6, FIG. 7 illustrates an example set of basic components of aportable computing device 700, such as the device 600 described withrespect to FIG. 6. In this example, the device includes at least oneprocessor 702 for executing instructions that can be stored in at leastone memory device or element 704. As would be apparent to one ofordinary skill in the art, the device can include many types of memory,data storage or computer-readable storage media, such as a first datastorage for program instructions for execution by the processor 702, thesame or separate storage can be used for images or data, a removablestorage memory can be available for sharing information with otherdevices, etc.

The device typically will include some type of display element 706, suchas a touch screen, electronic ink (e-ink), organic light emitting diode(OLED) or liquid crystal display (LCD), although devices such asportable media players might convey information via other means, such asthrough audio speakers. As discussed, the device in many embodimentswill include at least one image capture element 708, such as one or morecameras that are able to image a user, people, or objects in thevicinity of the device. In at least some embodiments, the device can usethe image information to determine gestures or motions of the user,which will enable the user to provide input through the portable devicewithout having to actually contact and/or move the portable device. Animage capture element also can be used to determine the surroundings ofthe device, as discussed herein. An image capture element can includeany appropriate technology, such as a CCD image capture element having asufficient resolution, focal range and viewable area, to capture animage of the user when the user is operating the device.

The device, in many embodiments, will include at least one audio element710, such as one or more audio speakers and/or microphones. Themicrophones may be used to facilitate voice-enabled functions, such asvoice recognition, digital recording, etc. The audio speakers mayperform audio output. In some embodiments, the audio speaker(s) mayreside separately from the device. The device, as described aboverelating to many embodiments, may also include at least one positioningelement 712 that provides information such as a position, direction,motion, or orientation of the device. This positioning element 712 caninclude, for example, accelerometers, inertial sensors, electronicgyroscopes, electronic compasses, and GPS elements.

The device can include at least one additional input device 718 that isable to receive conventional input from a user. This conventional inputcan include, for example, a push button, touch pad, touch screen, wheel,joystick, keyboard, mouse, trackball, keypad or any other such device orelement whereby a user can input a command to the device. These I/Odevices could even be connected by a wireless infrared or Bluetooth orother link as well in some embodiments. In some embodiments, however,such a device might not include any buttons at all and might becontrolled only through a combination of visual and audio commands suchthat a user can control the device without having to be in contact withthe device.

The example device also includes one or more wireless components 714operable to communicate with one or more portable computing deviceswithin a communication range of the particular wireless channel. Thewireless channel can be any appropriate channel used to enable devicesto communicate wirelessly, such as Bluetooth, cellular, or Wi-Fichannels. It should be understood that the device can have one or moreconventional wired communications connections as known in the art. Theexample device includes various power components 716 known in the artfor providing power to a portable computing device, which can includecapacitive charging elements for use with a power pad or similar deviceas discussed elsewhere herein. The example device also can include atleast one touch and/or pressure sensitive element 718, such as a touchsensitive material around a casing of the device, at least one regioncapable of providing squeeze-based input to the device, etc. In someembodiments this material can be used to determine motion, such as ofthe device or a user's finger, for example, while in other embodimentsthe material will be used to provide specific inputs or commands.

In some embodiments, a device can include the ability to activate and/ordeactivate detection and/or command modes, such as when receiving acommand from a user or an application, or retrying to determine an audioinput or video input, etc. In some embodiments, a device can include aninfrared detector or motion sensor, for example, which can be used toactivate one or more detection modes. For example, a device might notattempt to detect or communicate with devices when there is not a userin the room. If an infrared detector (i.e., a detector with one-pixelresolution that detects changes in state) detects a user entering theroom, for example, the device can activate a detection or control modesuch that the device can be ready when needed by the user, but conservepower and resources when a user is not nearby.

A computing device, in accordance with various embodiments, may includea light-detecting element that is able to determine whether the deviceis exposed to ambient light or is in relative or complete darkness. Suchan element can be beneficial in a number of ways. In certainconventional devices, a light-detecting element is used to determinewhen a user is holding a cell phone up to the user's face (causing thelight-detecting element to be substantially shielded from the ambientlight), which can trigger an action such as the display element of thephone to temporarily shut off (since the user cannot see the displayelement while holding the device to the user's ear). The light-detectingelement could be used in conjunction with information from otherelements to adjust the functionality of the device. For example, if thedevice is unable to detect a user's view location and a user is notholding the device but the device is exposed to ambient light, thedevice might determine that it has likely been set down by the user andmight turn off the display element and disable certain functionality. Ifthe device is unable to detect a user's view location, a user is notholding the device and the device is further not exposed to ambientlight, the device might determine that the device has been placed in abag or other compartment that is likely inaccessible to the user andthus might turn off or disable additional features that might otherwisehave been available. In some embodiments, a user must either be lookingat the device, holding the device or have the device out in the light inorder to activate certain functionality of the device. In otherembodiments, the device may include a display element that can operatein different modes, such as reflective (for bright situations) andemissive (for dark situations). Based on the detected light, the devicemay change modes.

Using the microphone, the device can disable other features for reasonssubstantially unrelated to power savings. For example, the device canuse voice recognition to determine people near the device, such aschildren, and can disable or enable features, such as Internet access orparental controls, based thereon. Further, the device can analyzerecorded noise to attempt to determine an environment, such as whetherthe device is in a car or on a plane, and that determination can help todecide which features to enable/disable or which actions are taken basedupon other inputs. If voice recognition is used, words can be used asinput, either directly spoken to the device or indirectly as picked upthrough conversation. For example, if the device determines that it isin a car, facing the user and detects a word such as “hungry” or “eat,”then the device might turn on the display element and displayinformation for nearby restaurants, etc. A user can have the option ofturning off voice recording and conversation monitoring for privacy andother such purposes.

In some of the above examples, the actions taken by the device relate todeactivating certain functionality for purposes of reducing powerconsumption. It should be understood, however, that actions cancorrespond to other functions that can adjust similar and otherpotential issues with use of the device. For example, certain functions,such as requesting Web page content, searching for content on a harddrive and opening various applications, can take a certain amount oftime to complete. For devices with limited resources, or that have heavyusage, a number of such operations occurring at the same time can causethe device to slow down or even lock up, which can lead toinefficiencies, degrade the user experience and potentially use morepower.

In order to address at least some of these and other such issues,approaches in accordance with various embodiments can also utilizeinformation such as user gaze direction to activate resources that arelikely to be used in order to spread out the need for processingcapacity, memory space and other such resources.

In some embodiments, the device can have sufficient processingcapability, and the imaging element and associated analyticalalgorithm(s) may be sensitive enough to distinguish between the motionof the device, motion of a user's head, motion of the user's eyes andother such motions, based on the captured images alone. In otherembodiments, such as where it may be desirable for the process toutilize a fairly simple imaging element and analysis approach, it can bedesirable to include at least one orientation determining element thatis able to determine a current orientation of the device. In oneexample, the at least one orientation determining element is at leastone single- or multi-axis accelerometer that is able to detect factorssuch as three-dimensional position of the device and the magnitude anddirection of movement of the device, as well as vibration, shock, etc.Methods for using elements such as accelerometers to determineorientation or movement of a device are also known in the art and willnot be discussed herein in detail. Other elements for detectingorientation and/or movement can be used as well within the scope ofvarious embodiments for use as the orientation determining element. Whenthe input from an accelerometer or similar element is used along withthe input from the camera, the relative movement can be more accuratelyinterpreted, allowing for a more precise input and/or a less compleximage analysis algorithm.

When using an imaging element of the computing device to detect motionof the device and/or user, for example, the computing device can use thebackground in the images to determine movement. For example, if a userholds the device at a fixed orientation (e.g. distance, angle, etc.) tothe user and the user changes orientation to the surroundingenvironment, analyzing an image of the user alone will not result indetecting a change in an orientation of the device. Rather, in someembodiments, the computing device can still detect movement of thedevice by recognizing the changes in the background imagery behind theuser. So, for example, if an object (e.g. a window, picture, tree, bush,building, car, etc.) moves to the left or right in the image, the devicecan determine that the device has changed orientation, even though theorientation of the device with respect to the user has not changed. Inother embodiments, the device may detect that the user has moved withrespect to the device and adjust accordingly. For example, if the usertilts their head to the left or right with respect to the device, thecontent rendered on the display element may likewise tilt to keep thecontent in orientation with the user.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 8 illustrates an example of an environment 800 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes an electronic client device 802, whichcan include any appropriate device operable to send and receiverequests, messages or information over an appropriate network 804 andconvey information back to a user of the device. Examples of such clientdevices include personal computers, cell phones, handheld messagingdevices, laptop computers, set-top boxes, personal data assistants,electronic book readers and the like. The network can include anyappropriate network, including an intranet, the Internet, a cellularnetwork, a local area network or any other such network or combinationthereof. The network could be a “push” network, a “pull” network, or acombination thereof. In a “push” network, one or more of the serverspush out data to the client device. In a “pull” network, one or more ofthe servers send data to the client device upon request for the data bythe client device. Components used for such a system can depend at leastin part upon the type of network and/or environment selected. Protocolsand components for communicating via such a network are well known andwill not be discussed herein in detail. Communication over the networkcan be enabled via wired or wireless connections and combinationsthereof. In this example, the network includes the Internet, as theenvironment includes a Web server 806 for receiving requests and servingcontent in response thereto, although for other networks, an alternativedevice serving a similar purpose could be used, as would be apparent toone of ordinary skill in the art.

The illustrative environment includes at least one application server808 and a data store 810. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to any deviceor combination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 808 caninclude any appropriate hardware and software for integrating with thedata store 810 as needed to execute aspects of one or more applicationsfor the client device and handling a majority of the data access andbusiness logic for an application. The application server providesaccess control services in cooperation with the data store and is ableto generate content such as text, graphics, audio and/or video to betransferred to the user, which may be served to the user by the Webserver 806 in the form of HTML, XML or another appropriate structuredlanguage in this example. The handling of all requests and responses, aswell as the delivery of content between the client device 802 and theapplication server 808, can be handled by the Web server 806. It shouldbe understood that the Web and application servers are not required andare merely example components, as structured code discussed herein canbe executed on any appropriate device or host machine as discussedelsewhere herein.

The data store 810 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 812 and userinformation 816, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 814. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 810. The data store 810 is operable,through logic associated therewith, to receive instructions from theapplication server 808 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on the user device 802. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 8. Thus, the depiction of the system 800 in FIG. 8should be taken as being illustrative in nature and not limiting to thescope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A computer-implemented method, comprising: undercontrol of one or more computer systems configured with executableinstructions, receiving image data; receiving audio data; determiningthat a first individual is represented in both the image data and theaudio data, by using a combination of facial recognition processing onthe image data, and voice recognition processing on the audio data; andcausing presentation of content, based at least on determining that thefirst individual is represented in both the image data and the audiodata.
 2. The computer-implemented method of claim 1, further comprising:generating authentication data corresponding to the first individualbased on the facial recognition processing on the image data.
 3. Thecomputer-implemented method of claim 1, further comprising: generatingauthentication data corresponding to the first individual based on thevoice recognition processing on the audio data.
 4. Thecomputer-implemented method of claim 1, further comprising: determiningenvironment information associated with a location of a computing devicethat is presenting the content; and determining the content based atleast in part on the environment information.
 5. Thecomputer-implemented method of claim 1, further comprising: receiving atleast one content preference; and associating the content preferencewith the first individual.
 6. The computer-implemented method of claim1, further comprising: generating authentication data for a user profileassociated with the first individual based on at least one of the audiodata or the image data.
 7. The computer-implemented method of claim 1,wherein the content includes at least one or more of video content,audio content, or image content.
 8. The computer-implemented method ofclaim 1, further comprising: determining an identity of a person inproximity of a computing device presenting the content based on one ormore of the image data, the audio data, or a presence of a deviceassociated with the person; and determining a user profile associatedwith the identity of the person.
 9. The computer-implemented method ofclaim 1, further comprising: determining at least a portion of thecontent is inappropriate based at least in part on a location of acomputing device and a user profile; and adjusting presentation of theat least the portion of the content.
 10. The computer-implemented methodof claim 9, wherein the user profile includes one or more restrictions,the method further comprising: determining the content based at least inpart on the one or more restrictions.
 11. A computing device,comprising: an image sensor capable of generating image data; an audiosensor capable of generating audio data; at least one processor; andmemory for storing instructions, the instructions, when executed by theprocessor, cause the computing device to: receive the image data fromthe image sensor; receive the audio data from the audio sensor;determine that a first individual is represented in both the image dataand the audio data, by using a combination of facial recognitionprocessing on the image data, and voice recognition processing on theaudio data; and cause presentation of content, based at least on thedetermination that the first individual is represented in both the imagedata and the audio data.
 12. The computing device of claim 11, whereinthe instructions, when executed by the processor, further cause thecomputing device to: generate authentication data corresponding to thefirst individual based on the facial recognition processing on the imagedata.
 13. The computing device of claim 11, wherein the instructions,when executed by the processor, further cause the computing device to:generate authentication data corresponding to the first individual basedon the voice recognition processing on the audio data.
 14. The computingdevice of claim 11, wherein the instructions, when executed by theprocessor, further cause the computing device to: determine environmentinformation associated with a location of the computing device; anddetermine the content based at least in part on the environmentinformation.
 15. The computing device of claim 11, wherein theinstructions, when executed by the processor, further cause thecomputing device to: receive at least one content preference; andassociate the content preference with the first individual.
 16. Thecomputing device of claim 11, wherein the instructions, when executed bythe processor, further cause the computing device to: generateauthentication data for a user profile associated with the firstindividual based on at least one of the audio data or the image data.17. The computing device of claim 11, wherein the content includes atleast one or more of video content, audio content, or image content. 18.The computing device of claim 11, wherein the instructions, whenexecuted by the processor, further cause the computing device to:determine an identity of a person in proximity of the computing devicebased on one or more of image data, audio data, or a presence of adevice associated with the person; and determine a user profileassociated with the identity of the person.
 19. The computing device ofclaim 11, wherein the instructions, when executed by the processor,further cause the computing device to: determine at least a portion ofthe content is inappropriate based at least in part on a location of thecomputing device and a user profile; and adjust presentation of the atleast the portion of the content.
 20. The computing device of claim 19,wherein the user profile includes one or more restrictions, theinstructions, when executed by the processor, further cause thecomputing device to: determine the content based at least in part on theone or more restrictions.